UNCERTAINTY SCORING PROJECT - README
=====================================

WHAT THIS PROJECT DOES
-----------------------
Scans earnings-call transcripts for a company for a keyword (currently just
"tariff") and tracks how often it comes up, per company and per quarter, to
proxy for tariff/USMCA-related uncertainty on earnings calls. Output is a
single Excel workbook with passage-level detail plus company, sector, and
S&P-index level trend tables and charts.

FOLDER LAYOUT
-------------
Uncertainty Scoring/
  transcript pdfs/          <- drop new Bloomberg transcript PDFs here
  transcripts/              <- converted .txt files land here (auto-named)
    pdf_to_txt.py
    count_transcripts.py
    index_sectors.xlsx      <- ticker -> sector -> S&P index
    Uncertainty Counts.xlsx <- the output workbook (overwritten each run)

HOW TO RUN
----------

  1. Download transcripts from the earnings calls found in Uncertainty Counts.xlsx in the "By Call" sheet. For this analysis, these were downloaded from            	Bloomberg.

  2. Convert new PDFs to text:
     python pdf_to_txt.py --input-dir "../transcript pdfs" --output-dir "../transcripts"

  3. Build/rebuild the workbook:
     python count_transcripts.py --input-dir "../transcripts" --output "Uncertainty Counts.xlsx" --sector-map index_sectors.xlsx

Step 2 regenerates the whole workbook from scratch every time - it is not
appended to incrementally, so it's safe to rerun after adding transcripts.

Before rerunning, worth checking that every ticker in the transcripts folder
also has a row in index_sectors.xlsx (otherwise it gets grouped under
"Unknown" sector/index instead of its real one).

ORGANIZING THE PDF INPUTS
--------------------------
No folder structure or consistent naming is needed - just drop all PDFs
flat into "transcript pdfs" (no subfolders; the script does not search
subfolders, only the top level of the input directory).

pdf_to_txt.py reads the ticker and call date straight out of the recurring
Bloomberg header on the page itself:
  FINAL TRANSCRIPT                                    2025-10-21
  General Motors Co (GM US Equity)
It does not rely on the source PDF's filename at all. Output .txt files are
auto-named TICKER_YYYYMMDD.txt regardless of what the PDF was called.

If a PDF is missing that header (or it doesn't match the expected pattern),
the script falls back to the PDF's own filename and prints a warning -
those should be checked and renamed to TICKER_YYYYMMDD.txt by hand.

Re-converting a PDF you've already converted is harmless - it just
overwrites the same output .txt (same ticker + date = same filename).

HOW THE KEYWORD COUNT WORKS
----------------------------
- Transcripts are split into paragraphs on blank lines (pdf_to_txt.py also
  treats a new speaker tag, e.g. "JOHN SMITH:" or "Q:", as a paragraph
  break, since Bloomberg PDFs don't always leave a clean blank-line gap
  between speakers).
- A paragraph "matches" if it contains the keyword ("tariff") anywhere,
  case-insensitive, as a substring.
- Paragraphs that look like an analyst's question (start with "Q -", "Q:",
  or "Q ") are detected and excluded. Only management/company-side commentary is counted.

WORKBOOK CONTENTS
------------------
- Passages: every matched (non-question) paragraph, one per row. This is removed in the replication package, given its proprietary nature.
- By Call: one row per company per call - match count, % of paragraphs,
  change vs. that company's prior call
- Company Trend: company x quarter matrix of match counts
- Company Comparison: lifetime totals/averages per company
- Sector Trend / Sector Comparison: same, rolled up by GICS sector
  (from index_sectors.xlsx)
- Overall Trend: market-wide totals and % of companies mentioning the
  keyword, by quarter, with charts
- Index Trend: Overall Trend split out by S&P index (500/400/600)
- Notes: keyword list and matching rules, for reference

Every .txt file in the transcripts folder gets a row on "By Call" even if
it has zero matches, so a company never silently drops out of a quarter.

OTHER
----------------------
- The use of "passages" to count the use of the keyword is an arbitrary choice. 
  A robustness check was run using total "tariff" wordcounts per call, 
  with similar results.
- Transcript files must end up named TICKER_YYYYMMDD.txt for the date to be
  correctly bucketed into a calendar quarter; anything else is grouped
  under an "unknown" period.
- The company universe is roughly the top 50 companies by market cap in
  each of the S&P 500/400/600 at the time the ticker list was built.
  Companies with irregular earnings-call schedules were excluded.
- If two calls for the same company land in the same calendar quarter
  (irregular schedule), Company Trend shows the max of the two counts for
  that cell and the script prints a warning to review manually.
- index_sectors.xlsx is the single source of truth for sector/index; a
  ticker missing from it still gets counted everywhere except the
  sector/index-specific sheets, where it's grouped under "Unknown".
